fix: storage v2 #5027

Maddiaa0 · 2024-03-07T11:20:50Z

Overview

Works around brillig blowup issue by altering the read and write opcodes to take in arrays of data. This is potentially just a short term fix.

Reading and writing to storage now take in arrays, code will not compile without this change, due to an ssa issue -> ISSUE
Tag checks on memory now just make sure the tag is LESS than uint64, rather than asserting that the memory tag is uint32, this should be fine.
We had to blow up the memory space of the avm to u64 as the entire noir compiler works with u64s. Arrays will not work unless we either
- Make the avm 64 bit addressable ( probably fine )
- Make noir 32 bit addressable ( requires alot of buy in ) chore: fix array access issue - nuclear edition #4814

subrepo: subdir: "noir" merged: "c44ef1484" upstream: origin: "https://github.com/noir-lang/noir" branch: "master" commit: "c44ef1484" git-subrepo: version: "0.4.6" origin: "???" commit: "???"

Maddiaa0 · 2024-03-07T11:24:27Z

barretenberg/cpp/src/barretenberg/flavor/generated/avm_flavor.hpp

@@ -45,7 +45,7 @@ class AvmFlavor {
    static constexpr size_t NUM_ALL_ENTITIES = 91;

    using Relations =
-        std::tuple<Avm_vm::avm_alu<FF>, Avm_vm::avm_main<FF>, Avm_vm::avm_mem<FF>, equiv_inter_reg_alu_relation<FF>>;
+        std::tuple<Avm_vm::avm_alu<FF>, Avm_vm::avm_mem<FF>, Avm_vm::avm_main<FF>, equiv_inter_reg_alu_relation<FF>>;


just regenerated these file to get around a conflict, they are just ordering changes ( as codegen is yet to be deterministic )
( nothing to see here ! )

AztecBot · 2024-03-07T12:40:25Z

Benchmark results

No metrics with a significant change found.

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit df089def and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	5,700	18,884	36,452
l1_rollup_calldata_gas	66,576	238,976	470,228
l1_rollup_execution_gas	194,524	500,282	909,486
l2_block_processing_time_in_ms	1,191	4,474	8,906 (+1%)
note_successful_decrypting_time_in_ms	208 (+2%)	542	992 (+1%)
note_trial_decrypting_time_in_ms	13.5 (+48%)	67.1 (-5%)	126 (+7%)
l2_block_building_time_in_ms	11,190 (-1%)	44,183 (-2%)	89,614 (+2%)
l2_block_rollup_simulation_time_in_ms	7,294 (-1%)	28,927 (-2%)	58,864 (+2%)
l2_block_public_tx_process_time_in_ms	3,867 (-2%)	15,189 (-2%)	30,601 (+2%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	13,833 (-5%)	26,880 (-3%)
note_history_successful_decrypting_time_in_ms	1,275 (-4%)	2,327 (-3%)
note_history_trial_decrypting_time_in_ms	93.9 (-24%)	96.5 (-25%)
node_database_size_in_bytes	18,780,240	35,573,840 (+1%)
pxe_database_size_in_bytes	29,923	59,478

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	250 (-2%)	44,736	28,001
private-kernel-ordering	194	52,625	14,627
base-rollup	655	177,932	997
root-rollup	68.4	4,320	825
private-kernel-inner	317 (-1%)	73,715	28,001
public-kernel-app-logic	170 (-1%)	32,254	25,379
merge-rollup	8.47 (-1%)	2,840	997

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	2 leaves	8 leaves	16 leaves	32 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves
batch_insert_into_append_only_tree_16_depth_ms	9.81	10.3 (-5%)	13.9 (+10%)	16.0 (-2%)	22.0 (-2%)	35.1 (-5%)	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.9	17.5	23.0	31.6	47.0	79.0	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.569	0.577 (-5%)	0.592 (+11%)	0.496 (-2%)	0.460 (-2%)	0.439 (-5%)	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	N/A	N/A	N/A	45.9	71.1 (-3%)	230 (-2%)	438 (-4%)	868 (-2%)	1,715
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	N/A	N/A	N/A	96.0	159	543	1,055	2,079	4,127
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.470	0.439 (-3%)	0.419 (-2%)	0.410 (-4%)	0.413 (-2%)	0.411
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	N/A	N/A	N/A	53.4	105 (-2%)	338 (+1%)	649 (-2%)	1,304	2,605 (+2%)
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	N/A	N/A	N/A	104	207	691	1,363	2,707	5,395
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	0.475	0.476 (-1%)	0.460 (+1%)	0.447 (-2%)	0.451	0.453 (+2%)
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	61.4 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	109	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	0.537 (+1%)	N/A	N/A	N/A	N/A	N/A	N/A

Miscellaneous

Transaction sizes based on how many contracts are deployed in the tx.

Metric	0 deployed contracts
tx_size_in_bytes	19,179

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	2,677 (-1%)	1,367 (-1%)

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	0.0340 (+14%)	475 (-1%)

## Overview Works around brillig blowup issue by altering the read and write opcodes to take in arrays of data. This is potentially just a short term fix. - Reading and writing to storage now take in arrays, code will not compile without this change, due to an ssa issue ->[ ISSUE ](AztecProtocol/aztec-packages#4979) - Tag checks on memory now just make sure the tag is LESS than uint64, rather than asserting that the memory tag is uint32, this should be fine. - We had to blow up the memory space of the avm to u64 as the entire noir compiler works with u64s. Arrays will not work unless we either - Make the avm 64 bit addressable ( probably fine ) - Make noir 32 bit addressable ( requires alot of buy in ) AztecProtocol/aztec-packages#4814 --------- Co-authored-by: sirasistant <[email protected]>

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-package: 0.26.6</summary> ## [0.26.6](aztec-package-v0.26.5...aztec-package-v0.26.6) (2024-03-08) ### Miscellaneous * **aztec-package:** Synchronize aztec-packages versions </details> <details><summary>barretenberg.js: 0.26.6</summary> ## [0.26.6](barretenberg.js-v0.26.5...barretenberg.js-v0.26.6) (2024-03-08) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>aztec-cli: 0.26.6</summary> ## [0.26.6](aztec-cli-v0.26.5...aztec-cli-v0.26.6) (2024-03-08) ### Features * Show bytecode size per function in CLI inspect-contract ([#5059](#5059)) ([cb9fdc6](cb9fdc6)) </details> <details><summary>aztec-packages: 0.26.6</summary> ## [0.26.6](aztec-packages-v0.26.5...aztec-packages-v0.26.6) (2024-03-08) ### Features * Basic public reverts ([#4870](#4870)) ([5cccc78](5cccc78)) * Deploying new inbox ([#5036](#5036)) ([fed729d](fed729d)) * Detect unknown note type ids in compute_note_hash ([#5086](#5086)) ([6206bec](6206bec)) * Easy deployment of protocol contracts in e2e ([#4983](#4983)) ([480161f](480161f)) * IPA documentation ([#4924](#4924)) ([48bd22e](48bd22e)) * Nullifier read requests in public kernel ([#4910](#4910)) ([0e44247](0e44247)) * Show bytecode size per function in CLI inspect-contract ([#5059](#5059)) ([cb9fdc6](cb9fdc6)) * Updating an SMT solver class ([#4981](#4981)) ([4b94d58](4b94d58)) ### Bug Fixes * Canonical contract address ([#5030](#5030)) ([b2af880](b2af880)) * Flaky deployment test ([#5035](#5035)) ([039eafc](039eafc)) * Pull the correct platform image for noir ([#5097](#5097)) ([3342371](3342371)) * Sleep function memory leak ([#5023](#5023)) ([a72cfea](a72cfea)), closes [#4817](#4817) * Storage v2 ([#5027](#5027)) ([fe3190e](fe3190e)) * Update protogalaxy cmake dependencies ([#5066](#5066)) ([507c374](507c374)) ### Miscellaneous * Address warnings in noir test suite ([#4966](#4966)) ([7ef4ef5](7ef4ef5)) * Bootstrap noir natively if nargo is invalid ([#5034](#5034)) ([df089de](df089de)) * Build avm transpiler if we are on mac ([#5039](#5039)) ([c2966b9](c2966b9)) * **ci:** Re-enable certain bb solidity ACIR tests ([#5065](#5065)) ([58e1ff4](58e1ff4)) * Cleanup of prover and verifier instances ([#4959](#4959)) ([f2fdefd](f2fdefd)) * Delete bootstrap scripts from `noir/noir-repo` ([#5044](#5044)) ([add91ca](add91ca)) * Disable `hello_world_example` noir test in aztec-packages CI ([#5061](#5061)) ([1be9243](1be9243)) * Join-split example Part 1 ([#4965](#4965)) ([b9de0f5](b9de0f5)) * Moving RootRollupInputs impl ([#5087](#5087)) ([f3d9f9b](f3d9f9b)) * Remove eccvm functionality to update the op queue and ensure ultra ops are populated through function ([#5084](#5084)) ([77954ab](77954ab)) ### Documentation * Parity circuit naming fixes ([#5076](#5076)) ([c255255](c255255)) </details> <details><summary>barretenberg: 0.26.6</summary> ## [0.26.6](barretenberg-v0.26.5...barretenberg-v0.26.6) (2024-03-08) ### Features * IPA documentation ([#4924](#4924)) ([48bd22e](48bd22e)) * Updating an SMT solver class ([#4981](#4981)) ([4b94d58](4b94d58)) ### Bug Fixes * Storage v2 ([#5027](#5027)) ([fe3190e](fe3190e)) * Update protogalaxy cmake dependencies ([#5066](#5066)) ([507c374](507c374)) ### Miscellaneous * **ci:** Re-enable certain bb solidity ACIR tests ([#5065](#5065)) ([58e1ff4](58e1ff4)) * Cleanup of prover and verifier instances ([#4959](#4959)) ([f2fdefd](f2fdefd)) * Join-split example Part 1 ([#4965](#4965)) ([b9de0f5](b9de0f5)) * Remove eccvm functionality to update the op queue and ensure ultra ops are populated through function ([#5084](#5084)) ([77954ab](77954ab)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- <details><summary>aztec-package: 0.26.6</summary> ## [0.26.6](AztecProtocol/aztec-packages@aztec-package-v0.26.5...aztec-package-v0.26.6) (2024-03-08) ### Miscellaneous * **aztec-package:** Synchronize aztec-packages versions </details> <details><summary>barretenberg.js: 0.26.6</summary> ## [0.26.6](AztecProtocol/aztec-packages@barretenberg.js-v0.26.5...barretenberg.js-v0.26.6) (2024-03-08) ### Miscellaneous * **barretenberg.js:** Synchronize aztec-packages versions </details> <details><summary>aztec-cli: 0.26.6</summary> ## [0.26.6](AztecProtocol/aztec-packages@aztec-cli-v0.26.5...aztec-cli-v0.26.6) (2024-03-08) ### Features * Show bytecode size per function in CLI inspect-contract ([#5059](AztecProtocol/aztec-packages#5059)) ([cb9fdc6](AztecProtocol/aztec-packages@cb9fdc6)) </details> <details><summary>aztec-packages: 0.26.6</summary> ## [0.26.6](AztecProtocol/aztec-packages@aztec-packages-v0.26.5...aztec-packages-v0.26.6) (2024-03-08) ### Features * Basic public reverts ([#4870](AztecProtocol/aztec-packages#4870)) ([5cccc78](AztecProtocol/aztec-packages@5cccc78)) * Deploying new inbox ([#5036](AztecProtocol/aztec-packages#5036)) ([fed729d](AztecProtocol/aztec-packages@fed729d)) * Detect unknown note type ids in compute_note_hash ([#5086](AztecProtocol/aztec-packages#5086)) ([6206bec](AztecProtocol/aztec-packages@6206bec)) * Easy deployment of protocol contracts in e2e ([#4983](AztecProtocol/aztec-packages#4983)) ([480161f](AztecProtocol/aztec-packages@480161f)) * IPA documentation ([#4924](AztecProtocol/aztec-packages#4924)) ([48bd22e](AztecProtocol/aztec-packages@48bd22e)) * Nullifier read requests in public kernel ([#4910](AztecProtocol/aztec-packages#4910)) ([0e44247](AztecProtocol/aztec-packages@0e44247)) * Show bytecode size per function in CLI inspect-contract ([#5059](AztecProtocol/aztec-packages#5059)) ([cb9fdc6](AztecProtocol/aztec-packages@cb9fdc6)) * Updating an SMT solver class ([#4981](AztecProtocol/aztec-packages#4981)) ([4b94d58](AztecProtocol/aztec-packages@4b94d58)) ### Bug Fixes * Canonical contract address ([#5030](AztecProtocol/aztec-packages#5030)) ([b2af880](AztecProtocol/aztec-packages@b2af880)) * Flaky deployment test ([#5035](AztecProtocol/aztec-packages#5035)) ([039eafc](AztecProtocol/aztec-packages@039eafc)) * Pull the correct platform image for noir ([#5097](AztecProtocol/aztec-packages#5097)) ([3342371](AztecProtocol/aztec-packages@3342371)) * Sleep function memory leak ([#5023](AztecProtocol/aztec-packages#5023)) ([a72cfea](AztecProtocol/aztec-packages@a72cfea)), closes [#4817](AztecProtocol/aztec-packages#4817) * Storage v2 ([#5027](AztecProtocol/aztec-packages#5027)) ([fe3190e](AztecProtocol/aztec-packages@fe3190e)) * Update protogalaxy cmake dependencies ([#5066](AztecProtocol/aztec-packages#5066)) ([507c374](AztecProtocol/aztec-packages@507c374)) ### Miscellaneous * Address warnings in noir test suite ([#4966](AztecProtocol/aztec-packages#4966)) ([7ef4ef5](AztecProtocol/aztec-packages@7ef4ef5)) * Bootstrap noir natively if nargo is invalid ([#5034](AztecProtocol/aztec-packages#5034)) ([df089de](AztecProtocol/aztec-packages@df089de)) * Build avm transpiler if we are on mac ([#5039](AztecProtocol/aztec-packages#5039)) ([c2966b9](AztecProtocol/aztec-packages@c2966b9)) * **ci:** Re-enable certain bb solidity ACIR tests ([#5065](AztecProtocol/aztec-packages#5065)) ([58e1ff4](AztecProtocol/aztec-packages@58e1ff4)) * Cleanup of prover and verifier instances ([#4959](AztecProtocol/aztec-packages#4959)) ([f2fdefd](AztecProtocol/aztec-packages@f2fdefd)) * Delete bootstrap scripts from `noir/noir-repo` ([#5044](AztecProtocol/aztec-packages#5044)) ([add91ca](AztecProtocol/aztec-packages@add91ca)) * Disable `hello_world_example` noir test in aztec-packages CI ([#5061](AztecProtocol/aztec-packages#5061)) ([1be9243](AztecProtocol/aztec-packages@1be9243)) * Join-split example Part 1 ([#4965](AztecProtocol/aztec-packages#4965)) ([b9de0f5](AztecProtocol/aztec-packages@b9de0f5)) * Moving RootRollupInputs impl ([#5087](AztecProtocol/aztec-packages#5087)) ([f3d9f9b](AztecProtocol/aztec-packages@f3d9f9b)) * Remove eccvm functionality to update the op queue and ensure ultra ops are populated through function ([#5084](AztecProtocol/aztec-packages#5084)) ([77954ab](AztecProtocol/aztec-packages@77954ab)) ### Documentation * Parity circuit naming fixes ([#5076](AztecProtocol/aztec-packages#5076)) ([c255255](AztecProtocol/aztec-packages@c255255)) </details> <details><summary>barretenberg: 0.26.6</summary> ## [0.26.6](AztecProtocol/aztec-packages@barretenberg-v0.26.5...barretenberg-v0.26.6) (2024-03-08) ### Features * IPA documentation ([#4924](AztecProtocol/aztec-packages#4924)) ([48bd22e](AztecProtocol/aztec-packages@48bd22e)) * Updating an SMT solver class ([#4981](AztecProtocol/aztec-packages#4981)) ([4b94d58](AztecProtocol/aztec-packages@4b94d58)) ### Bug Fixes * Storage v2 ([#5027](AztecProtocol/aztec-packages#5027)) ([fe3190e](AztecProtocol/aztec-packages@fe3190e)) * Update protogalaxy cmake dependencies ([#5066](AztecProtocol/aztec-packages#5066)) ([507c374](AztecProtocol/aztec-packages@507c374)) ### Miscellaneous * **ci:** Re-enable certain bb solidity ACIR tests ([#5065](AztecProtocol/aztec-packages#5065)) ([58e1ff4](AztecProtocol/aztec-packages@58e1ff4)) * Cleanup of prover and verifier instances ([#4959](AztecProtocol/aztec-packages#4959)) ([f2fdefd](AztecProtocol/aztec-packages@f2fdefd)) * Join-split example Part 1 ([#4965](AztecProtocol/aztec-packages#4965)) ([b9de0f5](AztecProtocol/aztec-packages@b9de0f5)) * Remove eccvm functionality to update the op queue and ensure ultra ops are populated through function ([#5084](AztecProtocol/aztec-packages#5084)) ([77954ab](AztecProtocol/aztec-packages@77954ab)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

sirasistant and others added 2 commits March 7, 2024 10:48

feat: avm storage

389f14a

subrepo: subdir: "noir" merged: "c44ef1484" upstream: origin: "https://github.com/noir-lang/noir" branch: "master" commit: "c44ef1484" git-subrepo: version: "0.4.6" origin: "???" commit: "???"

rebuild: avm

bedc4f9

Maddiaa0 commented Mar 7, 2024

View reviewed changes

fix: revert risky storage oracle changes

0949088

Maddiaa0 marked this pull request as ready for review March 7, 2024 11:41

Merge branch 'master' into md/storage

2f6e86b

Maddiaa0 requested a review from fcarreiro March 7, 2024 11:42

fix: restore missing test

e075c3b

Maddiaa0 mentioned this pull request Mar 7, 2024

fix(avm): go back to single value sstore / sloads, rather than arrays #5028

Closed

fcarreiro approved these changes Mar 7, 2024

View reviewed changes

Maddiaa0 added 5 commits March 7, 2024 12:00

fix: config yml

c4c3ce1

fix: end to end deps

befe4a0

fix: same for yarn project tests

65110dc

Merge branch 'md/fix-e2e-2-electric-boogaloo' into md/storage

ac777d6

Merge branch 'master' into md/storage

ee6178d

Maddiaa0 enabled auto-merge (squash) March 7, 2024 12:14

Maddiaa0 added 2 commits March 7, 2024 14:41

fmt

e47dac7

Merge branch 'master' into md/storage

a0d80c6

Maddiaa0 merged commit fe3190e into master Mar 7, 2024
93 of 94 checks passed

Maddiaa0 deleted the md/storage branch March 7, 2024 15:13

AztecBot mentioned this pull request Mar 7, 2024

chore(master): Release 0.26.6 #5037

Merged

AztecBot mentioned this pull request Mar 7, 2024

feat: Sync from aztec-packages noir-lang/noir#4494

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: storage v2 #5027

fix: storage v2 #5027

Maddiaa0 commented Mar 7, 2024 •

edited

Loading

Maddiaa0 Mar 7, 2024

AztecBot commented Mar 7, 2024 •

edited

Loading

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

fix: storage v2 #5027

fix: storage v2 #5027

Conversation

Maddiaa0 commented Mar 7, 2024 • edited Loading

Overview

Maddiaa0 Mar 7, 2024

Choose a reason for hiding this comment

AztecBot commented Mar 7, 2024 • edited Loading

Benchmark results

L2 block published to L1

L2 chain processing

Circuits stats

Tree insertion stats

Miscellaneous

Maddiaa0 commented Mar 7, 2024 •

edited

Loading

AztecBot commented Mar 7, 2024 •

edited

Loading